🖥️Server

Database design

The database is a Cloud Firestore Database. Cloud Firestore is a flexible, scalable NoSQL cloud database. It leverages Firebase Authentication to eliminate a middle tier and allow secure direct read/writes from the front end.

The development database can be found here. The production database can be found here.

The schema of the database is below.

Document	Field	Data
trusted_samples		All data for a particular trusted sample including metadata like site of collection, and user who entered the sample, as well as the results. Each document is a single sample that may contain multiple measurements.
	code: Str	The unique id associated with the sample
	created_by: Str	The id of the user who created the entry
	created_by_name: Str	The user name of the creator
	create_on: Str	The date which this sample was entered into the database
	n_wood:[float]	Percentage of nitrogen Valid Range: 0%..1%
	c_wood:[float]	Percentage of Carbon in Wood Valid Range: 40%...60%
	c_cel:[float]	Percentage of Carbon in Cellulose Valid Range: 40%...60%
	d13C_cel:[float]	The delta C-13 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: -35..-20
	d13C_wood:[float]	The delta C-13 isotope ratio as measured on wood by a mass spectrometer. Valid Range: -38...20
	d15N_wood:[float]	The delta N-15 isotope ratio as measured on wood by a mass spectrometer. Valid Range: -5...15
	d18O_wood: [float]	The delta O-18 isotope ratio as measured on wood by a mass spectrometer. Valid Range: 20...32
	d18O_cel:[float]	The delta O-18 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: 20...32
	last_updated: str	The datetime when this sample entry was last updated
	lat: Float	For trusted samples, it is the known latitude where the original tree had been felled for collection.
	lon: Float	For trusted samples, it is the known longitude where the original tree had been felled for collection.
	municipality: Str	The Brazilian municipality closest to the lat/lon given
	org: Str	The organization id that owns this data
	org_name: Str	The organization name
	points: [Struct]	Points is a collection of arbitrary fields as seen by an import. Every column during an import is loaded into the database into this document collection. Each row in this collection represents one timber subsample taken at a particular point in a larger timber sample.
	points[0..999].XXX	Each Field is a comprehensive list of values associated with a particular measured point on the timber sample. It will typically include the distance from center (with 0% being the center and 100% being the bark)
	site: Str	The name of the site where collection took place
	state: Str	The name of the state where collection took place.
	status: Str	Either "in_progress" or "complete" depending on whether the lab has finished processing the sample and values have been fully entered.
	trusted: Str	One of "trusted", "untrusted" or "unknown" Trusted samples are those where the original tree was sampled by a known reliable entity, such as USP, TNC or Imaflora. In this case, lat/lon should be filled in and represents the location where the entity took the sample from a live tree. Untrusted samples are those that may be seized by enforcement. In this case, lat/lon may be filled in, but represents a statement by the entity that cut the timber of the original tree location. Unknown is a subcategory of Untrusted and represents those pieces of timber where the lat/lon was not recorded or not claimed.
	visibility: Str	All current values should be private. In a future version, samples may be made "public" which allows users from other organizations to view them.
untrusted_samples		Identical to trusted_samples, but holds all samples where trusted = "untrusted"
unknown_samples		Identical to trusted_samples, but holds all samples where trusted = "unknown"
new_users / users		All data a new user enters when signing up for an account:
	date_requested	The date the user registered
	email: Str	Email
	name: Str	Name
	org: Str	TimberID Organization requested
	org_name: Str	Organization Name requested
	uid: Str	Unique User ID
organizations		Contains a list of documents with keys corresponding to each organization ID. Each document contains the members in the organization and their data (name, email, date joined), the org admins and their data (name, email, date joined), and the org name
	admins: [Struct]
	admins[].email: Str	admin email
	admins[].id	admin unique user id
	members: []	list of all members user emails
	org_email: Str	Email address for org
	org_name: Str	Name of org

Document

Field

Data

trusted_samples

All data for a particular trusted sample including metadata like site of collection, and user who entered the sample, as well as the results. Each document is a single sample that may contain multiple measurements.

code: Str

The unique id associated with the sample

created_by: Str

The id of the user who created the entry

created_by_name: Str

The user name of the creator

create_on: Str

The date which this sample was entered into the database

n_wood:[float]

Percentage of nitrogen Valid Range: 0%..1%

c_wood:[float]

Percentage of Carbon in Wood Valid Range: 40%...60%

c_cel:[float]

Percentage of Carbon in Cellulose Valid Range: 40%...60%

d13C_cel:[float]

The delta C-13 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: -35..-20

d13C_wood:[float]

The delta C-13 isotope ratio as measured on wood by a mass spectrometer.

Valid Range: -38...20

d15N_wood:[float]

The delta N-15 isotope ratio as measured on wood by a mass spectrometer. Valid Range: -5...15

d18O_wood: [float]

The delta O-18 isotope ratio as measured on wood by a mass spectrometer. Valid Range: 20...32

d18O_cel:[float]

The delta O-18 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: 20...32

last_updated: str

The datetime when this sample entry was last updated

lat: Float

For trusted samples, it is the known latitude where the original tree had been felled for collection.

lon: Float

For trusted samples, it is the known longitude where the original tree had been felled for collection.

municipality: Str

The Brazilian municipality closest to the lat/lon given

org: Str

The organization id that owns this data

org_name: Str

The organization name

points: [Struct]

Points is a collection of arbitrary fields as seen by an import. Every column during an import is loaded into the database into this document collection. Each row in this collection represents one timber subsample taken at a particular point in a larger timber sample.

points[0..999].XXX

Each Field is a comprehensive list of values associated with a particular measured point on the timber sample. It will typically include the distance from center (with 0% being the center and 100% being the bark)

site: Str

The name of the site where collection took place

state: Str

The name of the state where collection took place.

status: Str

Either "in_progress" or "complete" depending on whether the lab has finished processing the sample and values have been fully entered.

trusted: Str

One of "trusted", "untrusted" or "unknown" Trusted samples are those where the original tree was sampled by a known reliable entity, such as USP, TNC or Imaflora. In this case, lat/lon should be filled in and represents the location where the entity took the sample from a live tree. Untrusted samples are those that may be seized by enforcement. In this case, lat/lon may be filled in, but represents a statement by the entity that cut the timber of the original tree location. Unknown is a subcategory of Untrusted and represents those pieces of timber where the lat/lon was not recorded or not claimed.

visibility: Str

All current values should be private. In a future version, samples may be made "public" which allows users from other organizations to view them.

untrusted_samples

Identical to trusted_samples, but holds all samples where trusted = "untrusted"

unknown_samples

Identical to trusted_samples, but holds all samples where trusted = "unknown"

new_users / users

All data a new user enters when signing up for an account:

date_requested

The date the user registered

email: Str

name: Str

Name

org: Str

TimberID Organization requested

org_name: Str

Organization Name requested

uid: Str

Unique User ID

organizations

Contains a list of documents with keys corresponding to each organization ID. Each document contains the members in the organization and their data (name, email, date joined), the org admins and their data (name, email, date joined), and the org name

admins: [Struct]

admins[].email: Str

admin email

admins[].id

admin unique user id

members: []

list of all members user emails

org_email: Str

Email address for org

org_name: Str

Name of org

Editing/Viewing the database

You may edit and view the database contents using the Firebase Panel View. This allows you to view/edit data, run queries and examine security rules.

Security Rules

Firestore security rules are written using a non turing complete scripting language called "verax". You can find more information about the syntax along with examples here.

While we do not publish or document the security rules here, an example is shown below:

    match /trusted_samples/{documentId} {
      allow read: if canReadSample();
      allow write: if canWriteSample();
      allow delete, update: if canUpdateSample();
    }

The match statement makes the block apply to all document CRUD under the document path /trusted_samples. The {documentId} tag allows you to reference to specific document being requested for Read or Write.

The subsequent "allow X" allows those operations if the javascript-like expression returns true. In this case, several helper methods are called to evaluate whether the operation is allowed.

Cloud functions

Cloud functions will be used to give members and admins their permissions once they have been approved.

User journeys

Function Name	Trigger	Description
update-org-ee-auto-auth-sc	document.written	When a new user is registered to an organization, they are automatically permissioned in earth engine to access reference sample assets. This permission may also be used in general to download EE rasters without creating a separate EE account. Note: this requires a gmail address to have been registered.
fraud-detection-update-sample-sc	document.written	Performs analysis including origin validation when a sample is updated. This function performs all analysis sequentially, including "land use" "mapbiomas alerts" and origin verification
fraud-detection-new-isoscape-sc	Request	Reruns analysis after a new isoscape is created and stored in earth engine.
user-update-iam-check-sc	document.written	When a user is deleted by an Admin, this function cleans up the user account and fully deletes them.
fraud-detection-generate-maps	message published to Pub/Sub	This function creates or refreshes the stored maps shown in TimberID. More details

Function Name

Trigger

Description

update-org-ee-auto-auth-sc

document.written

When a new user is registered to an organization, they are automatically permissioned in earth engine to access reference sample assets. This permission may also be used in general to download EE rasters without creating a separate EE account. Note: this requires a gmail address to have been registered.

fraud-detection-update-sample-sc

document.written

Performs analysis including origin validation when a sample is updated. This function performs all analysis sequentially, including "land use" "mapbiomas alerts" and origin verification

fraud-detection-new-isoscape-sc

Request

Reruns analysis after a new isoscape is created and stored in earth engine.

user-update-iam-check-sc

document.written

When a user is deleted by an Admin, this function cleans up the user account and fully deletes them.

fraud-detection-generate-maps

message published to Pub/Sub

This function creates or refreshes the stored maps shown in TimberID. More details

Locally Debugging Cloud Functions

There is extensive online help on setting up your local cloud function debug environment. Once you have pip installed the framework, you can execute your function with:

functions-framework --target=YOUR_FUNCTION_NAME

If you have issues setting up the cloud function environment, you may also simply run the cloud function as its simply python code. The cloud functions in TimberID have alternate code paths for local authn which help in this regard.

Cloud Function Production Debugging and Logging

Extensive production and development environment information is found for each cloud function in the Google Cloud Console. Clicking on a function will bring you to the default function dashboard that shows invocation details, memory and performance statistics and recent errors.

Deploying Functions for Testing and Production

The github repository has github actions already configured to automatically deploy updated code for cloud functions when a PR is merged into the main branch. You see more details about this deployment in the yml file in github.

Merging code into the Main branch will automatically update the backend for development environment: test.timberid.org.

Before merging a PR to Main, you should have unit tests and manually test the function by running it directly or with the google cloud functions test harness.

Merging code into the PRD branch will automatically update the production environment.

Before merging code from Main into the PRD branch, you should run through the manual test procedures on test.timberid.org, which should have latest source from the Main branch.

Walkthrough Example: Adding data from terraclimate into the results page via a function.

Creating first user

If for some reason the database and user group is deleted or reset, to be able to start adding new users and organizations again, a site admin must be manually entered into the database with the following steps:

Go through the normal sign up process at timberid.org
Find the new user's User UID in the Firebase authentication tab
Create a new document in the "users" collection with the ID equal to the users UID
Enter a value for "name" and "org", and set the value of "role" to "site_admin". The values for name and org can be completely made up
This user can now log into TimberID using the account they created and have the ability to approve new organization and member requests

PreviousClient NextExternal integrations and maps

Last updated 9 months ago